feat: traceability metrics by FScholPer · Pull Request #484 · eclipse-score/docs-as-code

FScholPer · 2026-04-13T13:44:39Z

Frank Scholter Peres frank.scholter_peres@mercedes-benz.com, Mercedes-Benz Tech Innovation GmbH
Provider Information

github-actions · 2026-04-13T13:45:14Z

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run --lockfile_mode=error //src:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.4.2) and connecting to it...
INFO: Invocation ID: 61058728-a8cd-4a29-9af2-ddff10084c3b
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
WARNING: Target pattern parsing failed.
ERROR: Skipping '//src:license-check': no such target '//src:license-check': target 'license-check' not declared in package 'src' defined by /home/runner/work/docs-as-code/docs-as-code/src/BUILD
ERROR: no such target '//src:license-check': target 'license-check' not declared in package 'src' defined by /home/runner/work/docs-as-code/docs-as-code/src/BUILD
INFO: Elapsed time: 5.230s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

github-actions · 2026-04-13T13:49:05Z

The created documentation from the pull request is available at: docu-html

AlexanderLanin

see discussion in #485

FScholPer · 2026-04-16T08:42:49Z

closed #485 @AlexanderLanin can you please check again?

MaximilianSoerenPollak

Some comments for things I noticed.

Will take another look after lunch.

Overall looks like genuine good work that we can build upon.

MaximilianSoerenPollak · 2026-04-16T11:13:28Z

Also a general thing.

If there is any 'substantial' or partial generation from AI please add the disclaimer in the header.
According to 'eclipse' something to the lines of this should be fine:

# ╓                                                          ╖
# ║ Some portions generated by <AI Model Name>               ║
# ╙                                                          ╜

Model name meaning like 'Gpt' or 'Copilot' or 'Claude' etc.

FScholPer · 2026-04-16T13:44:09Z

Also a general thing.

If there is any 'substantial' or partial generation from AI please add the disclaimer in the header. According to 'eclipse' something to the lines of this should be fine:
# ╓                                                          ╖
# ║ Some portions generated by <AI Model Name>               ║
# ╙                                                          ╜
Model name meaning like 'Gpt' or 'Copilot' or 'Claude' etc.

I will add. But we should think about using tools like https://github.com/git-ai-project/git-ai in general

MaximilianSoerenPollak · 2026-04-16T14:18:17Z

I will add. But we should think about using tools like https://github.com/git-ai-project/git-ai in general

Probably might be worth a look to see if we could use that. I can bring it up in infra.

MaximilianSoerenPollak · 2026-04-16T14:25:44Z

@FScholPer I would propose that I can take these Ideas & Proposals that you have made in this PR and re-implement them into Docs-As-Code so they fit a better architecturally and makes sense with how testcases etc. actually behave & are created inside the DaC frameworks that deal with it.

I fear it would be a lot more work to merge this PR and then rebuild it / change it quiet heavily later when we want to integrate it.

Do you think this would make sense and is a good way to move forward with this issue?

@AlexanderLanin @a-zw please also advice here in your opinion what would be a good approach.

AlexanderLanin · 2026-04-17T11:03:34Z

to me it sounds like we need to discuss the goals first, then how to achieve it. Thats the reason why this mixes extensions with custom executables and no-one really understands why and how it should look like.

Co-authored-by: Maximilian Sören Pollak <maximilian.pollak@qorix.com> Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>

Co-authored-by: Andreas Zwinkau <95761648+a-zw@users.noreply.github.com> Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>

FScholPer · 2026-04-17T11:25:37Z

to me it sounds like we need to discuss the goals first, then how to achieve it. Thats the reason why this mixes extensions with custom executables and no-one really understands why and how it should look like.

Goals for where 1. check linking of requirements to code/tests in ci 2. add dashboards in all modules. Or what goals do you have in mind?

AlexanderLanin · 2026-04-17T15:56:09Z

@FScholPer we've discussed this PR, unfortunately we missed you. So here is our interpretation. Let's have a call next week.

The goals here are:

unify statistics/metrics across modules
provide thresholds for acceptable values, especially for CI
provide a dedicated executable / bazel target??

Here is an idea, which simplifies the design:

What do you think? (May be not self explanatory)
Do you want to give that a try?

Co-authored-by: Copilot <copilot@github.com>

FScholPer · 2026-05-11T15:11:01Z

Fixed the new comments. Mostly documentation issues

Co-authored-by: Alexander Lanin <Alexander.Lanin@etas.com> Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>

…e, source_linker config)

…lict)

AlexanderLanin

Note: This review was AI-assisted (Claude).

Summary

The architectural direction of this PR is solid and the iteration since the early feedback shows real progress: metrics are now computed inside Sphinx (single source of truth), exported as a well-defined metrics.json (schema-v1), and consumed by a clean gate script. The shared traceability_metrics.py module avoids duplication between dashboards and CI. The refactoring of standards.py to use generic_pie_* filters is a nice generalization that will pay off as more standards are onboarded. Test coverage for the new modules is good.

I'm organizing findings into three tiers so we can focus the remaining effort.

Must fix before merge

//:traceability_gate target does not exist for consumer repos. The docs() macro in docs.bzl does not create a traceability_gate target. The how-to guide (line 108) and commands.md both reference bazel run //:traceability_gate, but only //scripts_bazel:traceability_gate exists. Consumer repos that call docs() will get docs, docs_check, needs_json, etc. — but not traceability_gate. The macro should create an alias or wrapper so the documented command works out of the box. (See inline comments on commands.md and the how-to.)
metrics.json is an undeclared Bazel output. _write_metrics_json writes to Path(app.outdir) / "metrics.json" as a side effect of the Sphinx build, but docs.bzl doesn't declare it as an output of the :needs_json target. For bazel run this may work because the sandbox is relaxed, but for bazel build (which CI typically uses) the file may not be preserved or accessible at the documented path. This was flagged by @a-zw as well — needs explicit Bazel integration.

Should fix

Placeholder GitHub URL in plain_links mode. _render_code_link generates https://github.com/placeholder/placeholder/blob/unknown/… for Bazel sandbox builds. These fake URLs could propagate into needs.json and eventually into external-needs HTML. Consider using just the local path without a URL (the <> separator format already supports name-only rendering), or at minimum a clearly-sentinel scheme like local://.
Unrelated changes inflate the diff. The .pre-commit-config.yaml switch to tools/run_tool.sh wrappers, the tools/run_tool.sh script itself, and the shellcheck/actionlint aliases in BUILD are orthogonal to traceability metrics. Splitting them into a separate PR would make both easier to review and revert independently.
traceability_gate Bazel target pulls all_requirements but only uses stdlib. The gate script imports only argparse, json, os, sys, pathlib. The deps = all_requirements in scripts_bazel/BUILD adds ~30 pip packages for no reason, slowing down resolution and increasing the dependency surface.

Can be fixed later

process_requirements computed but not exported to metrics.json / gate. compute_traceability_summary returns process-requirement metrics and the dashboard shows them, but _write_metrics_json doesn't include them in the JSON output and the gate can't enforce thresholds on them. Fine for v1, but worth noting as a gap for future iterations.
Global mutable _DEFAULT_INCLUDE_EXTERNAL in traceability_dashboard.py. Using module-level state with set_default_include_external() works but is fragile for testing and any future multi-build scenario. A config-object approach would be cleaner.
CSS !important in score_design.css. box-shadow: none !important on .score-grid .sd-card overrides any consuming repo's card styling. This is fine for now but may need scoping to dashboard-specific containers later.

What's good

The metrics.json schema is well-designed: versioned, documented, and strict (additionalProperties: false). This will make evolution manageable.
Reusing compute_traceability_summary across dashboard filters and JSON export guarantees consistency — the numbers you see in the pie charts are the same ones CI gates on.
The gate script is clean and focused: it reads pre-computed data, prints a human-readable summary, and exits with distinct codes (0/1/2). Easy to integrate into any CI system.
Test coverage for the new code is good, especially the test_traceability_metrics.py and traceability_gate_test.py suites which cover edge cases like external needs, multiple types, and broken references.
The how-to guide (dashboards_and_quality_gates.rst) reads well and covers the right workflow: local → aggregated → CI rollout.

AlexanderLanin · 2026-05-12T19:45:11Z

+| `bazel run //:docs`                            | Builds documentation (also writes `metrics.json` via the score_metamodel extension)               |
 | `bazel run //:docs_check`                      | Verifies documentation correctness                                                                |
 | `bazel run //:docs_combo`                      | Builds combined documentation with all external dependencies included                             |
+| `bazel run //:traceability_gate -- --metrics-json bazel-bin/needs_json/_build/needs/metrics.json --min-req-code 70 --min-req-test 70 --min-req-fully-linked 60 --min-tests-linked 70` | Reads the pre-computed metrics.json and fails if coverage thresholds are not met                   |


AI-assisted review (Claude):

//:traceability_gate is documented here but no such target is defined in BUILD or created by the docs() macro in docs.bzl. Only //scripts_bazel:traceability_gate exists.

For the docs-as-code repo itself, add an alias in BUILD. For consumer repos, the docs() macro should create this target automatically — similar to how it creates //:docs_check and //:needs_json.

Must fix: The documented command won't work for anyone.

Fix-commit created by AI (Claude)

Fixed in 0d9195cb: docs() macro now creates a traceability_gate alias target for consumer repos.

AlexanderLanin · 2026-05-12T19:45:11Z

+.. code-block:: bash
+
+   bazel run //:docs && \
+   bazel run //:traceability_gate -- \


AI-assisted review (Claude):

Same issue: bazel run //:traceability_gate won't work for consumer repos because the docs() macro doesn't create this target. The how-to should also clarify the dependency: the gate needs metrics.json which is a side-effect of the Sphinx build, but Bazel doesn't declare it as an output — so bazel build //:needs_json && bazel run //:traceability_gate may not find the file.

Consider wiring the gate as a Bazel test target that depends on :needs_json, so bazel test //:traceability_gate does the right thing automatically.

Fix-commit created by AI (Claude)

Fixed in 0d9195cb: docs() macro now creates a traceability_gate alias. The how-to has also been updated to clarify the dependency on metrics.json from the Sphinx build (69254cfe).

AlexanderLanin · 2026-05-12T19:45:11Z

+    main = "traceability_gate.py",
+    visibility = ["//visibility:public"],
+    deps = all_requirements,
+)


AI-assisted review (Claude):

traceability_gate.py only uses stdlib modules (argparse, json, os, sys, pathlib). deps = all_requirements adds ~30 pip packages unnecessarily.

Suggested change

)

deps = [],

Fix-commit created by AI (Claude)

Fixed in 44dcc749: deps changed to [] — the gate script only uses stdlib modules.

AlexanderLanin · 2026-05-12T19:45:11Z

+def _render_code_link(plain_links: bool, metadata: RepoInfo, link: NeedLink) -> str:
+    if plain_links:
+        return (
+            "https://github.com/placeholder/placeholder/blob/unknown/"


AI-assisted review (Claude):

This hardcoded placeholder URL will end up in needs.json and could propagate into HTML when needs are imported as external needs by consuming repos. Users clicking these links would land on a 404.

Since plain_links mode is for sandbox builds where git info is unavailable, consider dropping the URL entirely and using just the display name:

return f"{link.file}:{link.line}"

The source_code_linker string-link regex already handles the name-only format (no <> separator).

Fix-commit created by AI (Claude)

Fixed in cba4fa8f: placeholder URL removed. Plain-links mode now renders as file:line without a fake GitHub URL.

The how-to and commands.md reference `bazel run //:traceability_gate` but the docs() macro never created this target. Consumer repos only had access to `//scripts_bazel:traceability_gate` which is an internal path. Add an alias so the documented command works out of the box.

traceability_gate.py only uses stdlib modules (argparse, json, os, sys, pathlib). Having deps = all_requirements added ~30 pip packages for no reason.

In Bazel sandbox builds (plain_links mode), _render_code_link generated a hardcoded https://github.com/placeholder/placeholder/blob/unknown/ URL that would 404 if clicked. Use just the file:line display format instead.

Adding a one-line .clwb entry does not constitute substantial AI generation warranting the Eclipse GenAI header.

Adding a py_binary target declaration is boilerplate, not substantial AI generation.

The condition `getEffectiveLevel() < 10` almost never triggers (only for NOTSET=0), so the function iterated all needs even at WARNING level where debug() output is discarded. Use isEnabledFor(DEBUG) instead.

…build

- docs_verify doesn't exist, the target is docs_check - remove leftover parenthetical from review comment

…_state docs

_write_metrics_json outputs include_external per type but the schema had additionalProperties: false without listing the field, so any schema validation would reject the generated JSON.

AlexanderLanin · 2026-05-12T23:17:40Z

Review fixes — batch update (AI-assisted, Claude)

Pushed 14 commits addressing findings from the May 11–12 review round. Individual reply comments have been added to each resolved thread.

Fixes

Bazel integration

docs() macro now creates //:traceability_gate alias for consumer repos (0d9195cb)
Removed unnecessary all_requirements pip deps from traceability_gate target — script uses only stdlib (44dcc749)

Documentation cleanup

How-to intro rewritten to concise purpose statement; setup section refers to :ref:setup`` instead of repeating (89d2ff08)
Corrected nonexistent docs_verify reference; cleaned up wording (89d2ff08)
Clarified that any docs build writes metrics.json; documented correct artifact paths (69254cfe)
commands.md: simplified docs description, hid internal extension name; user-facing targets use //: prefix (18d2c0d6)

Implementation state page

"Fully linked" now requires implementation (not just links); wording updated per suggestions (fb202b4a)
Needpie title → Tool Requirements Status; description labels updated (fb202b4a)

Source code linker

Removed hardcoded placeholder/placeholder GitHub URL from plain-links mode — now renders file:line only (cba4fa8f)
Fixed inverted debug log guard in _log_existing_links (be08bfce)

Housekeeping

AI disclaimers removed from .gitignore, scripts_bazel/BUILD, scripts_bazel/tests/BUILD (ec150c86, 85a9d261, fa04eda0)
Added missing include_external field to metrics JSON schema + gate test fixtures (2b4e5c4e, 2f65d264)
Added missing known_good parameter to docs() macro table (74ddf9b6)

Still open / not addressed here

requirement_types config — still documented as user-facing conf.py setting (AlexanderLanin flagged that types are already in metamodel; FScholPer asked whether to remove). Needs decision.
Two config values KNOWN_GOOD_JSON vs score_sourcelinks_json — purpose overlap unclear (AlexanderLanin's question unanswered).
AssertionError control flow — acknowledged tech debt, not a blocker.
Source code linker logic change (runs on both internal/external needs_json) — documented with rationale but impact assessment still pending.

…tion Bazel sandbox builds have no git metadata, so plain_links mode must still produce a URL that passes the metamodel ^https://github.com/.* pattern. The <> separator ensures sphinx-needs displays only the local path.

AlexanderLanin · 2026-05-12T23:32:18Z

Here are all my changes on one screen, please have a look: https://github.com/eclipse-score/docs-as-code/pull/484/changes/fcde17875d5ef3b72366b9ac61fa2a03ded36123..389675d6e6845eecf9593649b05dc4a303bc5f5b

FScholPer · 2026-05-13T06:55:16Z

Here are all my changes on one screen, please have a look: https://github.com/eclipse-score/docs-as-code/pull/484/changes/fcde17875d5ef3b72366b9ac61fa2a03ded36123..389675d6e6845eecf9593649b05dc4a303bc5f5b

Looks good for me

FScholPer requested review from AlexanderLanin, MaximilianSoerenPollak, dcalavrezo-qorix and nradakovic as code owners April 13, 2026 13:44

FScholPer linked an issue Apr 13, 2026 that may be closed by this pull request

Improvement: Introduce traceability coverage checker and CI reporting mode eclipse-score/score#2774

Open

github-project-automation Bot added this to INF - Infrastructure Community Apr 13, 2026

github-project-automation Bot moved this to Backlog in INF - Infrastructure Community Apr 13, 2026

a-zw suggested changes Apr 13, 2026

View reviewed changes

Comment thread scripts_bazel/traceability_coverage.py Outdated

Comment thread docs/how-to/test_to_doc_links.rst Outdated

Comment thread scripts_bazel/traceability_coverage.py Outdated

Comment thread scripts_bazel/traceability_coverage.py Outdated

AlexanderLanin requested changes Apr 14, 2026

View reviewed changes

MaximilianSoerenPollak reviewed Apr 16, 2026

View reviewed changes

a-zw suggested changes Apr 16, 2026

View reviewed changes

FScholPer and others added 7 commits April 17, 2026 11:07

feat(traceability): add coverage checker and reporting docs

15743fc

add coverage check

58ae80d

fix lint

4e9c60e

refactoring the coverage, metrics and dashboard

0ec5217

add generic filters

764da8d

Update src/extensions/score_metamodel/traceability_metrics.py

ec2e994

Co-authored-by: Maximilian Sören Pollak <maximilian.pollak@qorix.com> Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>

Apply suggestions from code review

a364257

Co-authored-by: Andreas Zwinkau <95761648+a-zw@users.noreply.github.com> Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>

FScholPer force-pushed the score-2774-traceability branch from 4585a27 to a364257 Compare April 17, 2026 11:08

readd genai headers

ecd1caf

FScholPer and others added 2 commits May 11, 2026 14:39

review fixes

ef1f5ca

Co-authored-by: Copilot <copilot@github.com>

added example

7745cce

Co-authored-by: Copilot <copilot@github.com>

AlexanderLanin reviewed May 11, 2026

View reviewed changes

AlexanderLanin changed the title ~~Score 2774 traceability~~ feat: traceability metrics May 11, 2026

FScholPer and others added 6 commits May 12, 2026 10:45

Update docs/reference/commands.md

0a1a885

Co-authored-by: Alexander Lanin <Alexander.Lanin@etas.com> Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>

fix: address PR #484 review comments (dashboards, implementation_stat…

15d8e91

…e, source_linker config)

Merge origin/main into score-2774-traceability (resolve docs.bzl conf…

5e49964

…lict)

fix: remove undocumented --fail-on-broken-test-refs flag from docs

56779d3

format fix

5403782

Merge branch 'main' into score-2774-traceability

fcde178

AlexanderLanin reviewed May 12, 2026

View reviewed changes

AlexanderLanin added 14 commits May 12, 2026 23:39

fix: remove unnecessary pip deps from traceability_gate target

44dcc74

traceability_gate.py only uses stdlib modules (argparse, json, os, sys, pathlib). Having deps = all_requirements added ~30 pip packages for no reason.

fix: remove fake placeholder URL from plain_links code rendering

cba4fa8

In Bazel sandbox builds (plain_links mode), _render_code_link generated a hardcoded https://github.com/placeholder/placeholder/blob/unknown/ URL that would 404 if clicked. Use just the file:line display format instead.

fix: remove AI disclaimer from .gitignore

ec150c8

Adding a one-line .clwb entry does not constitute substantial AI generation warranting the Eclipse GenAI header.

fix: remove AI disclaimer from scripts_bazel/BUILD

85a9d26

Adding a py_binary target declaration is boilerplate, not substantial AI generation.

fix: correct debug log guard in _log_existing_links

be08bfc

The condition `getEffectiveLevel() < 10` almost never triggers (only for NOTSET=0), so the function iterated all needs even at WARNING level where debug() output is discarded. Use isEnabledFor(DEBUG) instead.

fix: clarify CI quality gate docs — metrics.json comes from any docs …

69254cf

…build

fix: correct nonexistent docs_verify reference and clean up wording

89d2ff0

- docs_verify doesn't exist, the target is docs_check - remove leftover parenthetical from review comment

fix: simplify docs command description — hide internal extension name

18d2c0d

fix: clarify "fully linked" requires implementation in implementation…

fb202b4

…_state docs

fix: add missing include_external field to metrics JSON schema

2b4e5c4

_write_metrics_json outputs include_external per type but the schema had additionalProperties: false without listing the field, so any schema validation would reject the generated JSON.

fix: add include_external to gate test fixtures for schema compliance

2f65d26

fix: add missing known_good parameter to docs() macro table

74ddf9b

fix: remove AI disclaimer from scripts_bazel/tests/BUILD

fa04eda

Conversation

FScholPer commented Apr 13, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License Check Results

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexanderLanin left a comment

Choose a reason for hiding this comment

Uh oh!

FScholPer commented Apr 16, 2026

Uh oh!

MaximilianSoerenPollak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaximilianSoerenPollak commented Apr 16, 2026

Uh oh!

FScholPer commented Apr 16, 2026

Uh oh!

MaximilianSoerenPollak commented Apr 16, 2026

Uh oh!

MaximilianSoerenPollak commented Apr 16, 2026

Uh oh!

AlexanderLanin commented Apr 17, 2026

Uh oh!

FScholPer commented Apr 17, 2026

Uh oh!

AlexanderLanin commented Apr 17, 2026

Uh oh!

FScholPer commented May 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexanderLanin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Summary

Must fix before merge

Should fix

Can be fixed later

What's good

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 13, 2026 •

edited

Loading

AlexanderLanin left a comment •

edited

Loading